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ABSTRACT 
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success in completing tasks or overall completion time. Use of other 
navigation aids built into the browser, such as the "home" and "back" 
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usability. No differences among models were found here either. Results of 
this study tentatively indicate that the three navigation models, as tested, 
performed equally well and further that designers should give fair 
consideration to each of the models pending the purpose and audience of the 
site being designed. Further research is needed, since several potentially 
confounding factors were identified. (Author/MES) 
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Abstract 



2 This stud y investigated performance differences between three different web-based navigation models: 

vo linear, persistent, and semi-persistent menu structures. Forty-four college students were placed into one of 

^ ^ three navigation conditions and completed information-finding tasks. No significant differences were 

Q found among the models with respect to success in completing tasks or overall completion time. Use of 

W other navigation aids built into the browser such as the HOME and BACK buttons was also measured, as 

well user satisfaction and perceived usability. No differences among models were found here either. 
Results of this study tentatively indicate that the three navigation models, as tested, performed equally well 
and further that designers should perhaps give fair consideration to each of the models pending the 
purpose and audience of the site being designed. Further research is needed, since several potentially 
confounding factors were identified. 

Introduction 

Everyday thousands of people around the world make it a part of their day to locate some piece of 
information on the World Wide Web (WWW). Sometimes users of the WWW are browsing, simply looking at 
various web sites to find something that interests them, while other users access the WWW with a specific task in 
mind. 

Web sites vary a great deal in terms of the audience being served and the purpose of the content being 
provided. There are sites that target children, adolescents and adults; educators, business executives, and 
researchers; vacationers, television junkies, and sports fanatics. The list goes on as long as there are topics and 
people who are interested in them. Of all these sites that are currently in use, some are intended to be used 
occasionally or even rarely, while there are other sites, such as corporate Intranet sites, that are used daily for very 
specific, job-critical tasks. 

Regardless of the usage context for a particular web site, all web sites must employ some kind of navigation 
scheme to aid the user in traveling from one page to another. Designers of web sites currently enjoy little in the way 
of published, empirically supported guidelines specific to navigation issues. Site navigation is an essential element 
of an effective web site and there is a pressing need for researchers to begin investigating the issues and questions 
surrounding web-based navigation models. Most sites today are still built based either on personal experience and 
taste, an incompatible medium (i.e., paper), or at best on usability data with limited generalizability. 

The purpose of this study was to investigate the effectiveness of three popular navigational models used in 
on the WWW. Specifically this study had hoped to begin laying the foundation for future studies dealing with web 
sites and how people perform using them. 

Review of Related Literature 

Though the research base on navigating within a WWW environment is limited, there are a number of 
studies which have been done in the last twenty years that have important implications for designing navigation 
schemes for web-based information systems. The varying research areas of influence include studies done on 
paper-based navigation, pre-web hypertext models, and web site usability. Hierarchical and persistent menu 
strategies on the web are a relatively new area of interface design interest. Only recently have researchers begun to 
address the question of menu development as applied to a hypertext environment, and in particular the World Wide 
Web. Empirical research is just beginning to emerge, mostly in the form of dissertations and research-oriented 
courses and seminars. 
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Effectiveness and Challenges of Hierarchical Information Structures 

Research done by Chang (1995) indicates that hierarchical structures result in the most efficient navigation 
and information retrieval. In designing specific hierarchical menu structures, depth (the number of levels) and 
breadth (the number of items per page) were shown to be prime considerations by Harney (1990). Other factors that 
have been shown to be of importance in hierarchical menu design include: 

1. menu dimension, 

2. task complexity, and 

3. the knowledge structure of the WWW user (Jacko, 1994). 

It has been shown that WWW users experience difficulty navigating large information systems. McKnight, 
Dillon, and Richardson (1990) provided evidence that users can become disoriented during hypertext searching 
tasks. In this study, as in many information spaces, the challenge for the user is to know where they are in the 
system, where they want or need to go to in the system, and finally to somehow keep track of where they have been 
in the system. Disorientation results when users can not accomplish those tasks. It has been proposed that hypertext 
designs can be improved by providing structural cues to the WWW user, thereby facilitating user orientation (Rouet, 
et al. 1996). 

One of the few recommendations one can locate in the research comes from Sano (1997) where he states 
that an effective and intuitive online information structure has two critical characteristics. First, the information 
which users need the most should be within two to three levels away from their starting point. Second, grouping of 
related items is essential to avoiding long lists of information that the user must scroll through. 

Structural Cues 

Paper documents have long used a wide array of navigation aids to help users find the information they 
need. Tables of contents have been shown to be effective navigation aids in paper documents. Indices, vertical tabs, 
and other devices have also been used effectively in paper-based documents. 

Many web-based information systems continue to use individual vertical lists of links for each level in a 
hierarchy. Frick, et al. (1995) and Corry, et al. (1997) found that these lists worked well in meeting user 
expectations and resulted in high navigation performance on a university web site. In the same design, exemplars 
were added to each link to aid the user in understanding more about what each link would really provide them. 

Users in this study reported that these exemplars were helpful. Spool (1997) also reports that link 
descriptors have been shown in usability testing to be extremely effective aids in helping users to decide whether the 
link is in fact what they want or not. 

Another navigation tool used extensively in online systems is the expanding/contracting hierarchy. Such 
lists of choices have been shown to be effective in the Windows operating system, as well as in the Apple Computer 
OS. Early hypertext systems such as Superbook have also been shown to use expanding/contracting outlines 
successfully (Egan, et al. 1989). Specific to the web. Nation, et al. (1997) conducted an evaluation of an 
expandable/contractible table of contents tool called WebTOC. The tool is a Java applet that displays a dynamic 
hierarchy on the left of the screen with the content to the right side. Early results show that the applet can assist users 
in navigating web sites. Lastly, expanding/contracting tables of contents have also been shown to decrease browsing 
times when compared with fully expanded versions (Chimera et al. 1994). 

Another navigation aid that is popular on the web is what we call hierarchical chains. These chains are 
structural cues in that they communicate to the user exactly where in the hierarchy they are and where they are 
currently. Usually a horizontal line of hypertext items separated by colons or arrows, each item in the chain is a level 
in the hierarchy that is hot-linked, allowing the user to return with a single click to any previously visited level. 
Examples of this navigation strategy can be seen at two popular and heavily usability-tested web sites. They are: 

http://www.vahoo.com 

http://www.useit.com . 

Nielsen has usability tested this method and found it to be an effective aid in helping users navigate 
web-based information systems. 

Additional Design Considerations 

After determining the overall hierarchical design, other issues that need to be addressed concern types of 
menu structure (embedded versus explicit) and sub-menu design. Some research indicates that embedded menus 
produce more accurate searches, permit fewer screens viewed, and are preferred overall by users (Koved & 
Shneiderman, 1986). Menu design can also be divided into three levels: alphabetical, categorical, and random. In 
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terms of overall efficiency for novice users t there is no significant difference between alphabetical and categorical 
(Coll, Coll, and Nandavar, 1993). 

Another consideration for web designers concerns the type of tasks that need to be supported. Navigation 
aids in a hypertext environment should be developed according to the tasks of the user. In terms of information 
search tasks, navigation via indexes seems to be the preferred method of users (Edwards and Hardman, 1989, cited in 
Smith, 1997). Developers should also provide information to the user informing them of their current progress, such 
as a path history (Jones, Farquhar and Surry, 1995). In terms of sub-menu design, studies by Scheurman and Peck 
(1991) indicate that users of sub-menus that provide a return to mam menu option, versus a return to previous 
sub-menu option, spend less time on sub-menus and therefore are more efficient. 

Other issues of importance that need to be considered are usability and its measurement, which have 
included various factors in the past such as: time to completion, number of errors, and deviations from optimal paths 
(Smith, 1994, cited in Smith, 1997). Other considerations could include finding required information, identifying the 
meaning of a link, and obtaining the required information when it is requested (Smith, 1997). 

There are currently a number of different navigation strategies that designers implement in web-based 
information systems. Some of the many navigation options available to web designers include: 

■ plain hypertext, often in the form of lists, categories, hierarchies (e.g. http J/www. indiana.edu/iub/) 

frames that provide persistent or perpetual navigation (e.g. http://www.macromedia.com and http J! www.steelcase.com) 

■ expandable/collapsible hierarchies (Danny Goodman's site) 

■ forms, that scroll and/or drop down lists of linkable items (e.g. http-7/www. pw.com) 

■ hierarchical chains (e.g. http Jt www.yahoo.com, and http://www.useit.com) 



Some of the navigation methods listed above have been tested with users both in usability settings and in 
empirical studies. We have described the outcomes of those efforts. Other methods have yet to be tested at all, or at 
least in any published format. 

Methods 
Research Design 

The methodology used in this study was a classical experimental design with subjects randomly assigned to 
one of three treatment conditions. These three treatments were three web sites, which contained same information 
but different navigation menus. 



■ Condition A: non-persistent (changing) hierarchical menus displayed as vertical lists of choices with non-hot exemplars to 
the right. 

Condition B: A + persistent top-level menu displayed as vertical list of choices in the left column of the screen, and 

■ Condition C: A + semi -persistent horizontal hierarchical lists of choices at the top of the screen that represent the path 
taken down the hierarchy (textual map). 

These navigation models can be viewed at URL: http://education.indiana.edu/-frick/aect99/designs.html. 

The dependent variables in this study were the subjects performance on the task component of the study, the 
number of times the user voluntarily used the back button, and each subject’s attitude. Attitude variables were 
constructed to collect subjective data about (a) appeal of the designs, (b) their usability, and (c) perceptions of 
"getting lost" in the web sites. 

Pilot Tests 

Six pilot tests were completed prior to actual data collection. The pilot tests were used to determine the 
usability of data collection instruments, to determine the difficulty of the tasks, and to ensure that data collection 
procedures worked satisfactorily. 

Instruments tested during pilot testing included a background questionnaire, task list, and the attitude 
survey. The results of the pilot tests were used to ensure the background and the attitude questionnaires were 
understandable and usable. The pilot test was also used to finalize the task list. Over 30 tasks were tested during the 
pilot tests. The difficulty and usability of each task was determined based on the whether the subjects found the 
correct answer and the time it took to find the answer. The easiest tasks were deleted from the task list, and 24 tasks 
were selected for use in the actual study. 

Instruments 

The background questionnaire consisted of five questions about the subjects web and computer experiences. 
The task instrument consisted of 24 items. The task items were standard usability tasks that users were asked in 
order to find specific information in the web site. The tasks were evenly distributed among the web site. The attitude 
questionnaire was an instrument consisting of 18 Likert-type questions and one open-ended question for general 
comments. The 18 items consisted of positively and negatively worded statements about the navigation model used. 
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which were intended to measure its: (a) appeal, (b) usability, and (c) "getting lost" issues. In response to each 
statement, participants indicated which one of five ordered responses, ranging from Astrongly agree@ to Astrongly 
disagree @. 

Prior to individual sessions with each subject, the researcher needed to set up the computer and browser for 
the test. The monitor screen resolution was set to 640 X 480. The browser was opened and maximized. The 
Adirectory buttons @ were disabled. The URL was set to the treatment tested in that session. The AHOME@ button 
on the navigation bar was set to the Home Page URL of the treatment. The Amemory cache @ and Adisk cache @ 
were set to zero so that web server logs would contain navigation moves from page to page. The IP number of the 
computer workstation was recorded for later web server log analysis. Finally, the link history list was reset. 

Subjects 

Potential participants were screened beforehand to help determine who would be appropriate for the 
research study. The major criterion that was used to select subjects was little or no prior experience with the Indiana 
University School of Education web site. Subjects were excluded who were already familiar with this web site, since 
prior knowledge might confound the results. The resulting 44 subjects were undergraduate and graduate students 
and public school teachers, who were randomly assigned to the three treatment conditions ( n = 14, 15, 15 
respectively). 

After completing the background questionnaire, each subject was asked to find answers to the 24 
information tasks (e.g.. What is the phone number of the External Relations Office? Find the Web page for the 
Education Library. Find information pertaining to Graduate Admissions.). The tasks were randomly ordered for 
each subject, who was given up to three minutes to find the answer. When the subject believed she had found the 
answer, she notified the researcher. If the answer was not found within three minutes, the subject gave up, or if the 
subject came to the wrong place in the web site, it was counted as an incorrect response. The researcher also 
recorded task completion time, and navigation buttons used. 

Immediately following the completion of the 24 task questions, each subject was asked to complete the 
attitude questionnaire. Most subjects completed the experiment in 45 minutes to an hour. 

Results 

Reliabilities of the attitude scales were estimated by Cronbach’s alpha for appeal (0.90), usability (0.89) and 
getting lost (0.70). The number of completed tasks, the total time for completing the tasks, the total number of 
voluntary back button clicks, and the attitude scales were analyzed by using the one-way Avon’s. The statistical 
significance level used for hypothesis testing was set a priori to .05. 

There was no significant difference among group means on number of tasks completed successfully. 



Table 1. A NOVA for the number of tasks completed successfully 



Design 


Mean 


SD 


F 


Sig. 


A (n= 14) 


19.29 


2.16 


.069 


.934 


B (n = 15) 


19.20 


2.43 






C(n= 15) 


18.93 


3.30 







There was no significant difference among the three designs on the total amount of time to do all 24 tasks. 
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Table 2. AN OVA for the total task completion time in seconds. 



Design 


Mean 


SD 


F 




A 


1249.71 


226.16 


1.407 


.256 


B 


1194.65 


271.70 






C 


1388.63 


435.64 







There was no significant difference among the three designs on the number of voluntary uses of the back button. 
Table 3. ANOVA for the number of voluntary back button clicks 



Design 


Mean 


SD 


F 


Sig. 


A 


18.92 


14.31 


2.997 


.062 


B 


7.64 


6.89 






C 


16.86 


15.69 







Note that the significance level (.062) approaches the .05 level, and it is possible that Design B (persistent top-level menu along the left side) 
results in fewer uses of the BACK button, compared to the other navigation models. 



There was no significant difference among groups with respect to the appeal of the navigation model. 
Table 4. ANOVA for subjective perception of appeal 



Design 


Mean 


SD 


F 


Sig. 


A 


12.43 


2.56 


.369 


.694 


B 


13.07 


3.71 






C 


13.53 


3.93 







There was no significant difference among groups with respect to the perceived usability of the navigation model. 
Table 5. ANOVA for subjective perception of usability 



Design 


Mean 


SD 


F 


Sig. 


A 


9.21 


1.89 


.163 


.832 


B 


9.20 


3.08 






C 


9.73 


3.45 







There was no significant difference among groups with respect to the perception of Agetting lost@ in the navigation 
model. 

Table 6. ANOVA for subjective perception of Agetting lost@ 



Design 


Mean 


SD 


F 


Sig. 


A 


13.36 


1.39 


.841 


.832 


B 


13.53 


2.72 






C 


13.93 


3.33 
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Discussion 



We should emphasize that the web site for the School of Education (which is similar to Design A in our 
study, with its bottom-of-page navigation system removed) was previously developed and revised through extensive 
usability testing, making it as easy as possible for users to find answers to typical information seeking tasks such as 
those in our study. The persistent or semi-persistent menus, which were added in Designs B and C in our study, 
apparently did not significantly enhance its perceived usability or appeal, which was very good from the start. With 
web sites where little or no usability testing is used in the design process, we can speculate that persistent or semi- 
persistent menu systems might make a bigger difference. 

We strongly believe that more research is needed before we can be sure that no differences exist among 
these different navigational models, because several usability factors became apparent during the study. Factors that 
pertained to a majority of respondents are discussed below, and each of these behaviors was either observed by the 
research team or specifically mentioned by the subjects. 

First, most subjects were not initially cognizant of the menu schemes in Designs B and C. In Design B , after 
six to seven questions subjects typically started to use the sidebar menu scheme that was provided in the left column. 
Most subjects mentioned that they did not notice the sidebar menu function until after the tasks had started. Also, it 
took answering a few questions before the subjects understood how the side bar in Design B was applicable to their 
task-searching behavior. In Design C, several subjects mentioned that they did not notice the hierarchical menu at 
the top until they had navigated the structure for several levels. It took considerably more time for subjects in 
Design C to notice and begin to use the menu chain at the top of the page versus the time required for subjects in 
Design B with the sidebar at the left of the screen. One exception to the time factor in Design B was a subject who 
was a media specialist who immediately understood and consistently used the menu. 

A majority of subjects mentioned that the exemplars were the most beneficial aspect of navigation. Design 
A contained exemplars for all its menus. Designs B and C contained Design A with the addition of menu choices on 
the side or top, respectively, which did not contain exemplars due to limitations in screen real estate. We can 
speculate that if users visit a site frequently, they would learn the meanings of the labels used for the menu items and 
be less dependent on exemplars for deciding which part of the hierarchy they need to go to. Our study was relatively 
short (less than an hour), and subjects were screened to ensure that they were not familiar with its structure. 

In addition to the exemplar confound, most subjects used the home button for navigation at the beginning of 
each task. As part of each treatment, the home button was set to go to the home page for that design. For most 
subjects, it was much easier to navigate to the home page using the home button and then to use the exemplars to 
find specific pieces of information. This would be atypical for most web sites, and hence may have confounded our 
results. 

Most subjects mentioned that during Design B the sidebar menu should have been contained on the home 
page as well as the other pages in the document. Subjects stated that having the sidebar menu on sub-levels and not 
on the main level was confusing. They did not understand how the side bar menu was helpful until they discovered 
it, typically after six to seven questions. 

Subjects were confused by the dead links. In order to keep subjects from leaving the experimental web 
designs used in the study, links to web pages outside the navigation models were disabled intentionally to prevent 
confounding by other inconsistent navigation schemes used on external pages. It was not practical to reproduce the 
entire School of Education web site for each of the three treatment conditions, which would have consisted of 
literally thousands of web pages developed by different departments and units. Consequently, several subjects gave 
answers to the task they were completing based on the link name with exemplars itself instead of clicking and 
following the link. 




Conclusion 

In summary, the results were somewhat disappointing in that we expected to find designs B and C to be 
more efficient with respect to time to find answers to information seeking tasks. If choices are visually present to 
navigate by means of shortcuts to other parts of a menu hierarchy, then this increases the compactness of the 
hypermedia system (i.e., shorter distances between nodes). It also increases the flexibility, since users have multiple 
ways to move around a site. However, our data from this study do not support these expectations (with the exception 
that Design B may decrease reliance on the use of the back button for navigation, since the differences approached 
statistical significance). One could conclude that there are indeed no differences among the three types of navigation 
models, and hence could use other design criteria for making a choice among them. For example, if screen space is 
needed to minimize scrolling, then traditional non-persistent menu structures are more efficient with respect to use of 
screen real estate. The purpose of the web site might also dictate which model is preferable. For example, Yahoo is 
a large clearinghouse for links to the web (a Ayellow pages@ function). For such a large, deep hierarchy of choices, 
the use of semi-persistent hierarchical chains (or textual maps, as represented in our Design Q occupies less screen 
real estate than a sidebar menu and yet provides flexibility for moving around the hierarchy. 
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We suggest that future research should be done, given constraints we faced in this study and the possibility 
of confounded results due to those constraints. 
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